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Introduction 

Autism  spectrum  disorders  (ASDs)  are  disorders  of  the  brain  that  affect  as  many  as  1  in  88 
children.  Without  intensive  and  appropriate  treatment,  the  long-term  outcomes  for  children  with 
ASDs  remain  bleak  and  are  associated  with  a  high  divorce  rate  among  parents  and  increased 
risk  for  mental  health  disorders  among  family  members.  The  efficacy  of  and  empirical  support 
for  interventions  for  ASDs  based  on  applied  behavior  analysis  (ABA)  is  well  documented. 
Unfortunately,  these  services  are  often  not  available  to  military-dependent  children  because  there 
are  not  enough  appropriately  trained  individuals  to  design  and  provide  ABA  services.  This  project 
will  demonstrate  how  web-based  technologies  can  increase  the  availability  of  effective  treatment 
for  military-dependent  children  with  ASDs.  By  evaluating  a  technology-enhanced  treatment 
delivery  model,  military  families  will  be  able  to  receive  empirically  supported  treatment  services 
in  a  timely  manner  anywhere  in  the  world.  Also,  training  therapists  to  implement  ABA  programs 
using  a  web-based  model  will  greatly  increase  the  number  of  well-trained  therapists  in  areas 
around  many  military  bases. 
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Body 

Task  2.  Complete  parent  and  tutor  training  and  EIBI  aims  for  5  cohorts,  which  includes  50 

participant  families  and  their  ABA  tutors  (timeframe,  Months  12-44). 

From  Statement  of  Work;  Tasks  include  three  distinct  sub-tasks  for  each  cohort.  Our  training  protocol 
based  on  E-Learning  using  the  latest  web-based  and  televideo-based  instruction  will  provide  an 
efficient  and  effective  mechanism  for  training  military  parents  of  children  with  autism  anywhere  in  the 
world  by  experts  in  one  location  (University  of  Nebraska  Medical  Center  in  Omaha).  These  trained 
parents  will  then  be  able  to  implement  effective  behavior  management  and  teaching  strategies  with 
high  procedural  integrity  (90%  accuracy).  Second,  we  will  use  the  protocol  we  developed  to  train 
adults  to  become  ABA  tutors  and  to  implement  early  intervention  procedures  with  high  integrity  (90% 
accuracy)  in  areas  of  the  world  where  such  services  are  unavailable.  The  training  protocol  (using  web- 
based  E-Learning  instructional  methods)  will  be  the  same  for  the  parents  and  the  ABA  tutors,  but  they 
will  have  more  extensive  curricula  (with  the  curriculum  for  the  ABA  tutors  being  more  comprehensive). 
Third,  we  will  evaluate  changes  in  cognitive,  language,  social,  play,  and  adaptive  skills  and  decreases 
in  problem  behaviors  among  children  with  autism  in  military  families  who  receive  technology-enhanced 
early  intervention  services  that  are  supervised  by  our  experts.  The  children  in  the  technology- 
enhanced  early  intervention  treatment  group  will  show  significantly  greater  improvements  than  children 
randomly  assigned  to  a  waitlist-control  group. 

In  each  cohort,  there  will  be  10  children  with  autism  recruited  with  at  least  one  parent  (cohort  n  => 
10)  and  one  ABA  tutor  (cohort  n  =>  10)  per  child.  In  the  first  cohort,  5  of  the  children  and  their 
corresponding  parents  (n  =>  5)  and  ABA  tutors  (n  =>  5)  will  be  randomly  assigned  to  the  technology- 
enhanced  early  intervention  treatment  group  and  the  other  5  children  and  their  corresponding  parents 
(n  =>  5)  and  ABA  tutors  (n  =>  5)  will  be  assigned  to  the  waitlist-control  group. 


We  have  reported  progress  for  Experiments  T  2,  &  3  in  this  order. 


1 .  Randomized  Clinical  Trial:  Recruit,  pretest,  and  train  10  parents  in  the  technology-enhanced 
test  group  and  10  parents  in  the  waitlist-control  group.  Schematic  1  details  the  progress  for 
the  technology-enhanced  parent-training  evaluation  (Experiment  1). 

Schematic  1. 


I  Parent  Training 
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a.  We  have  initial  outcome  data  for  the  between-subjects  comparison  with  three  parents 
that  were  randomized  to  the  technology-enhanced  group  and  five  parents  that  were 
randomized  to  the  waitlist-control  group.  The  difference  in  the  numbers  across  the 
groups  is  due  to  attrition,  as  denoted  in  the  schematic.  In  addition,  two  parents,  who 
were  assigned  to  the  test  group,  are  progressing  through  the  technology-enhanced 
training  while  three  other  parents  have  been  assigned  to  the  waitlist-control  group.  We 
will  continue  to  enroll  parents  as  we  enroll  families  for  the  early-intervention  services  in 
Experiment  3.  In  Figures  1  and  2,  we  present  the  initial  outcome  data  for  the  parents 
who  have  completed  Experiment  1  to  date. 

In  Figure  1 ,  the  type  of  assessment  is  depicted  on  the  x-axis.  The  top  panel  shows  the 
participants’  performance  for  the  Behavioral  Implementation  Skills  for  Work  Activities 
(BISWA)  assessment  and  the  bottom  panel  shows  the  participants’  performance  for  the 
Behavioral  Implementation  Skills  for  Play  Activities  (BISPA).  The  gray  and  white  bars 
denote  the  average  performance  across  all  participants  for  the  test  and  waitlist-control 
groups,  respectively;  the  open  circles  denote  the  individual  performance  for  each 
participant  in  each  group.  The  percentage  of  trials  implemented  correctly  is  depicted  on 
the  y-axis,  and  this  measure  was  calculated  by  dividing  the  total  number  of  trials 
implemented  correctly  plus  the  total  number  of  errors,  including  both  errors  of 
commission  (implementing  a  skill  incorrectly;  e.g.,  delivering  reinforcement  following  an 
incorrect  response)  and  errors  of  omission  (failing  to  implement  a  skill  when  required; 
e.g.,  omitting  descriptive  praise  following  a  correct  response).  This  method  of  scoring 
provided  an  overall  measure  of  how  accurately  the  tutors  carried  out  the  procedures, 
regardless  of  the  unequal  number  of  programmed  opportunities  across  the  types  of 
confederate  responses. 

Prior  to  experiencing  the  technology-enhanced  training,  all  of  the  parents’  performance 
was  unsatisfactory  across  both  primary  dependent  measures  and  comparable  across 
groups,  with  relatively  worse  performance  for  the  BISPA.  Following  the  technology- 
enhanced  training,  the  preliminary  results  with  the  three  parents  show  robust 
improvements  in  performance.  All  parents  implemented  the  skills  correctly  on  more  than 
80%  of  the  trials  on  the  BISWA  and  BISPA  assessments  (gray  bar  on  posttest  column); 
by  contrast,  the  performance  of  the  five  parents  in  the  control  group  showed  no 
improvement  (white  bar  on  posttest  column). 

In  Figure  2,  instead  of  reporting  the  percentage  of  trials  implemented  correctly  on  the  y- 
axis,  the  percentage  of  skills  implemented  correctly  for  each  component  skill  (e.g., 
delivers  behavior-specific  instructions)  is  depicted.  This  method  of  scoring  provided 
information  on  which  specific  skills  the  parents  were  implementing  with  high  fidelity  and 
which  ones  should  be  targeted  for  additional  training,  and  this  method  controlled  for 
differences  in  the  number  of  programmed  opportunities  across  the  types  of  confederate 
responses. 

As  observed  in  Figure  1,  notable  differences  were  observed  for  the  three  parents  who 
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received  the  technology-enhanced  training  for  the  BISWA  (top  panel)  and  BISPA  (bottom 
panel),  and  no  improvement  was  observed  for  the  five  parents  assigned  to  the  waitlist- 
control  group.  We  plan  to  apply  preliminary  statistical  tests  after  at  least  four  parents 
have  matriculated  through  the  training  in  each  group.  These  data  should  be  obtained  by 
October  1®*,  2014. 


Figure  1. 


Figure  2. 
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2.  Tutor-training  Evaluation  (Experiment  2) 

a.  Randomized  Clinical  Trial:  Recruit,  pretest,  and  train  10  tutors  in  the  technology- 
enhanced  test  group  and  10  tutors  in  the  waitlist-control  group.  Schematic  2  details  the 
progress  for  the  technology-enhanced  tutor-training  evaluation. 


Schematic  2. 


Tutor  Training 


-©•  Randomized  Clinical  Trial 

— ©  Completed 


Test  =  6 
Control  =  7 


Enrolled 


- ©Test  =  0 


■  Attrition-5 


- 0  Control  =  0 


■  Attrition-3 


©  Completing  Consent  &  Baseline 


2 

■  Attrition  - 1 


©  Single-Subject  Design 


Attrition:  3 


i.  Efficacy  Outcomes:  Since  last  year’s  annual  report,  13  participants  have  completed 
the  randomized  clinical  trial.  As  a  result,  we  have  initial  outcome  data  for  the  between- 
subjects  comparison  with  six  participants  who  were  randomly  assigned  to  the  test 
group  and  received  the  technology-enhanced  tutor  training  and  seven  participants  who 
were  assigned  to  the  waitlist-control  group  and  did  not  receive  training.  We  have 
conducted  preliminary  statistical  analyses  with  the  first  4  test-control  dyads  to  complete 
the  evaluation  (i.e.,  the  first  8  participants). 

The  performances  of  the  four  participants  in  the  test  and  control  groups  are  depicted 
as  group  averages  in  Figure  3  below,  and  the  results  from  the  preliminary  statistical 
tests  are  summarized.  The  type  of  assessment  is  depicted  on  the  x-axis.  The  right 
panel  shows  the  participants’  performance  for  the  BISWA  assessment  and  the  left 
panel  shows  the  participants’  performance  for  BISPA.  The  closed  and  open  data  plots 
show  the  mean  performance  for  participants  in  the  treatment  and  control  group, 
respectively.  The  top  panel  reports  the  percentage  of  trials  implemented  correctly  and 
the  bottom  panel  reports  the  percentage  of  skills  implemented  correctly  for  each 
component  skill.  The  method  for  calculating  these  measures  is  that  same  as 
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described  above  for  the  parent-training  evaluation. 

Both  groups  performed  poorly  on  the  BISPA  during  the  pretest  (Figure  3,  upper-left 
panel),  although  the  treatment  group  showed  a  slightly  higher  mean  percentage  of 
correct  trials  than  did  the  control  group  (Ms  =  25.8%  and  8.3%  on  the  pretest  for  the 
treatment  and  control  groups,  respectively).  However,  the  difference  between  the 
means  for  the  treatment  and  control  groups  on  the  posttest  was  notably  larger  (Ms  = 
88.5%  and  13.3%  on  the  posttest  for  the  treatment  and  control  groups,  respectively). 
Due  to  the  observed  difference  between  the  groups  on  the  pretest,  we  analyzed  the 
data  with  a  general  linear  model  using  performance  on  the  pretest  as  a  covariable. 
When  statistically  controlling  for  the  effects  of  the  pretest,  the  difference  between  the 
treatment  and  control  group  means  on  the  posttest  was  large  (i.e.,  partial  eta  squared 
of  .765,  which  is  equivalent  to  a  Cohen’s  d  of  3.6)  and  statistically  significant  (F  =  13.0; 
p  <  .05). 

Similar  results  were  obtained  for  the  percentage  of  trials  implemented  correctly  on  the 
BISWA,  which  are  depicted  in  the  upper-right  panel  of  Figure  3.  The  treatment  group 
showed  a  slightly  higher  mean  percentage  of  correct  trials  than  did  the  control  group 
on  the  BISWA  pretest  (Ms  =  50.3%  and  28.5%  for  the  treatment  and  control  groups, 
respectively),  but  the  difference  was  notably  larger  on  the  posttest  (Ms  =  97.5%  and 
43.3%  for  the  treatment  and  control  groups,  respectively).  When  statistically 
controlling  for  the  effects  of  the  pretest,  the  difference  between  the  treatment  and 
control  group  means  on  the  posttest  was  large  (i.e.,  a  partial  eta  squared  of  .766; 
which  is  equivalent  to  a  Cohen’s  d  of  3.6)  and  statistically  significant  (F  =  13.1;  p  < 

.05). 

The  bottom-left  panel  of  Figure  3  shows  the  mean  percentage  of  component  skills 
mastered  for  the  BISPA  across  the  treatment  and  control  groups.  Both  groups 
displayed  mastery  performance  on  a  small  percentage  of  the  skills  (Ms  =  8.3%  and  0% 
on  the  pretest  for  the  treatment  and  control  groups,  respectively).  By  contrast,  the 
treatment  group  had  mastered  the  majority  of  the  component  skills  by  the  posttest, 
whereas  the  control  showed  little  improvement  (Ms  =  66.8%  and  8.3%  on  the  posttest 
for  the  treatment  and  control  groups,  respectively).  When  statistically  controlling  for 
the  effects  of  the  pretest,  the  difference  between  the  treatment  and  control  group 
means  on  the  posttest  was  large  (i.e.,  partial  eta  squared  of  .708;  which  is  equivalent 
to  a  Cohen’s  d  of  3.1)  and  statistically  significant  (F  =  12.1;  p  <  .05). 

The  bottom-right  panel  of  Figure  3  depicts  the  results  for  the  mean  percentage  of 
component  skills  mastered  on  the  BISWA.  Both  groups  displayed  mastery 
performance  on  a  small  proportion  of  the  component  skills  (Ms  =  25.0%  and  6.8%  on 
the  pretest  for  the  treatment  and  control  groups,  respectively).  By  contrast,  the 
treatment  group  had  mastered  almost  all  of  the  component  skills  by  the  posttest, 
whereas  the  control  showed  only  slight  improvement  (Ms  =  93.3%  and  22.5%  on  the 
posttest  for  the  treatment  and  control  groups,  respectively).  When  statistically 
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controlling  for  the  effects  of  the  pretest,  the  difference  between  the  treatment  and 
control  group  on  the  posttest  was  large  (i.e.,  partial  eta  squared  of  .97;  which  is 
equivalent  to  a  Cohen’s  d  of  11.4)  and  statistically  significant  (F  =  128.1;  p  <  .001). 

The  significant  interaction  between  the  pretest  and  group  membership  suggests  that 
the  tutors  in  the  treatment  group  who  scored  relatively  higher  on  pretest  tended  to 
show  a  better  response  to  the  treatment.  These  data  were  recently  published  in 
Research  in  Autism  Spectrum  Disorders  (5-year  Impact  Factor,  2.665)  in  their  latest 
issue  (Fisher,  Luczynski,  Flood,  Lesser,  Machado,  &  Pizza,  2014). 

Since  the  preliminary  analyses  with  the  first  eight  participants,  two  and  three  additional 
participants  have  matriculated  through  the  test  and  control  groups,  respectively. 

Similar  results  have  been  obtained  for  these  participants. 

Figure  3. 

Behavioral  Implementation  Skills  Behavioral  Implementation  Skills 

for  Play  Activities  (BISPA)  for  Work  Activities  (BISWA) 


Pretest  Posttest  Pretest  Posttest 


Assessments 
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ii.  Social  Validity  Outcomes:  Following  completion  of  the  technology-enhanced  training, 
tutors  are  asked  to  provide  closed-  and  open-ended  responses  (ratings  and  open- 
ended  comments)  regarding  their  acceptability  of  the  web-based  technology,  the  type 
of  content  covered  during  training,  interactions  with  the  consultant  and  scheduling,  and 
overall  impression  of  the  training  program.  In  Table  1  below,  we  report  results  for  five 
tutors  who  have  completed  the  questionnaire.  The  tutors  used  a  7-point  Likert  scale 
with  the  following  ratings:  7  =  strongly  agree,  4  =  no  opinion,  1  =  strongly  disagree. 

The  mean  satisfaction  rating  with  the  web-based  technology  was  5.6  (range,  4.4  to  6), 
with  the  least  satisfaction  expressed  for  the  use  of  Blackboard  as  the  medium  for 
accessing  the  e-learning  modules  and  the  most  satisfaction  expressed  for  the  audio 
and  visual  quality  during  role-play  sessions.  The  mean  satisfaction  rating  with  the 
content  of  the  virtual  training  program  was  6.7  (range,  6.4  to  7),  with  high  satisfaction 
expressed  for  all  aspects  of  the  training  content,  including  the  type  of  information 
covered,  the  role-play  component,  the  quality  and  organization  of  the  e-learning 
modules,  and  the  amount  learned  during  training.  The  mean  satisfaction  rating  for  the 
interactions  with  the  researcher  (behavioral  consultant)  was  6.8  (range,  6.6  to  7),  with 
high  satisfaction  expressed  for  all  aspects  of  the  interactions,  including  the  consultant’s 
ability  to  answer  questions,  the  interactions  with  the  consultant,  the  option  of 
completing  the  training  at  the  technician’s  pace,  and  the  flexibility  offered  in  scheduling 
meetings  and  role  plays.  Finally,  the  mean  rating  for  overall  satisfaction  with  the 
training  was  6.8  (range,  6.8  to  7),  and  all  of  the  tutors  indicated  that  they  would 
recommend  the  training  to  others. 

Based  on  these  results,  the  only  area  of  our  technology-enhanced  model  that  has 
received  lower  satisfaction  ratings  is  the  use  of  Blackboard  as  the  software  platform  to 
remotely  deliver  our  training  modules  to  tutors  throughout  training.  Several  tutors 
(participants  2,  4,  &  5)  have  reported  a  variety  of  technical  glitches  with  accessing, 
downloading,  and  viewing  the  modules.  In  response,  we  are  considering  the  use  of 
other  platforms  to  deliver  the  modules.  Following  a  six-month  beta  test  of  secure 
cloud-based  platforms,  the  University  of  Nebraska  Medical  Center  has  recently 
adopted  Box  (https://www.box.com)  as  a  secure  content-sharing  platform  for  sharing 
information  in  clinical  practice  and  research  applications.  We  plan  to  use  Box  to  share 
the  modules  for  two  participants  to  determine  whether  the  reported  technical  issues 
can  be  eliminated. 


Table  1 

Social  Validity  Assessment  and  Results  for  Technology-Enhanced  Tutor  Training 

Questions 


Tutor 

Ratings _ 

2  3  4  5  M 


Web-Based  Teehnology 

7  7  5  4  7  6.0 
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1 .  Prior  to  interacting  with  my  consultant  and  conducting  virtual  meetings,  you  were 
asked  to  setup  some  technology.  I  was  satisfied  with  the  process  of  accessing  the 
GotoMeeting  or  Adobe  Connect  meeting  room  and  starting  my  audio  and  webcam? 

2.  We  conducted  virtual  meetings,  using  either  Gotomeeting  or  Adobe  Connect,  to 
review  the  goals  for  the  upcoming  week  and  provide  a  status  update.  I  was  satisfied 
with  the  reliability  (e.g.,  consistent  internet  connection)  and  the  audio  and  video 
quality  of  these  virtual  meetings. 


7  6  5  4  7  5.8 


3.  Virtual  meetings  were  also  conducted  for  teaching  during  role-plays.  I  was  satisfied 

with  the  reliability  (e.g.,  consistent  internet  connection)  and  the  audio  and  video  7  6  6  5  7  6.2 

quality  during  the  role-plays. 


4.  We  provided  the  modules  and  quizzes  through  Blackboard’s  software  platform.  I  was 
satisfied  with  Blackboard  as  the  method  to  deliver  the  modules  and  quizzes. 

Content  Covered  during  Training 

5.  Each  module  covered  the  core  content  of  a  given  topic.  I  was  satisfied  with  the 
amount  and  type  of  information  covered  in  the  modules. 


5  4  6  3  4  4.4 


7  6  7  7  6  6.6 


6.  We  conducted  several  role-plays  throughout  the  training  and  each  role-play  covered  a 


different  topic.  I  was  satisfied  with  the  role  plays  as  a  component  of  training  (i.e.,  an 
opportunity  to  practice  the  skills  and  receive  real-time  feedback). 

7 

7 

7 

6 

6 

6.6 

7.  I  was  satisfied  with  the  quality  and  organization  of  these  modules? 

7 

7 

7 

6 

5 

6.4 

8.  I  was  satisfied  with  the  amount  that  I  learned  during  the  training. 

7 

7 

7 

7 

7 

7 

Interactions  with  your  Consultant  and  Scheduling 

9.  During  role-plays  and  throughout  the  training  program,  you  had  the  opportunity  to 
ask  your  consultant  questions  about  the  content  of  the  curriculum  or  rationale  for 
certain  skills  during  the  role  plays.  I  was  satisfied  with  my  consultant’s  ability  to 
answer  my  questions. 

7 

7 

7 

7 

7 

7 

10.  I  am  satisfied  with  the  interactions  that  I  had  with  my  consultant  (e.g.,  time  to  respond 
to  questions,  weekly  updates). 

7 

7 

7 

7 

7 

7 

1 1 .  The  training  program  was  arranged  so  that  you  could  proceed  at  your  own  pace.  I 
was  satisfied  with  option  of  completing  the  program  at  my  own  pace 

7 

7 

6 

7 

6 

6.6 

12.  I  was  satisfied  with  the  flexibility  of  scheduling  meetings  and  role  plays. 

7 

7 

6 

7 

6 

6.6 

Overall  Training  Program 


1 3 .  Overall,  I  was  satisfied  with  this  program. 

7 

7 

7 

7 

6 

6.8 

14.  Would  you  recommend  this  type  of  web-based  instruction  to  other  individuals  who 

Y 

Y 

Y 

Y 

Y 

Yes 
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are  not  able  to  receive  on-site  instruction? 


b.  Single-Subject  Design.  For  tutors  that  withdrew  from  the  randomized  clinical  trial,  we  are 
evaluating  the  effects  of  the  technology-enhanced  training  within  a  single  subject 
experimental  design.  This  evaluation  is  currently  being  conducted  with  three  tutors. 
Within  this  design,  confidence  that  the  technology-enhanced  training  is  responsible  for 
the  improvement  in  the  tutors’  post-training  performance  will  be  established  by  showing 
that  increases  in  BISWA  and  BISPA  only  occurs  when  the  training  is  implemented 
following  different  baseline  lengths  for  each  tutor.  This  experimental  design  controls  for 
common  threats  to  internal  validity  (e.g.,  history,  testing;  Campbell  &  Stanley,  1963).  The 
results  will  be  interpreted  using  structured  visual  inspection  procedures  developed  in  our 
lab  that  are  highly  reliable  (inter-observer  agreement  =  .95)  and  valid  (correlation  with  an 
expert  panel,  r  =  .94). 

We  included  this  design  to  account  for  participants  who  were  randomly  assigned  to 
waitlist-control  group  and  attrition  occurred  for  his  or  her  matched  participant  in  the 
technology-enhanced  group.  Instead  of  asking  the  participant  to  continue  waiting  until 
another  participant  completes  the  training,  the  participant  will  be  enrolled  in  the  multiple- 
baseline  design.  The  use  of  both  designs  will  increase  our  retention  of  participants  by 
being  sensitive  to  their  total  time  enrolled  in  the  study  while  also  showing  experimental 
control  over  the  training  curriculum.  Two  of  three  tutors  have  completed  the  pretests,  and 
one  tutor  has  nearly  completed  the  technology-enhanced  training.  After  they  finish  the 
training,  posttest  measures  will  be  collected  and  analyzed. 


2.  Recruit,  pretest,  and  train  and  treat  10  children  in  the  technology-enhanced  test  group  and  10 
children  in  the  waitlist-control  group.  Schematic  3  details  the  progress  for  the  technology- 
enhanced  early-intervention  services  for  children  with  autism  (Experiments  3). 


Schematic  3. 


Child  Early  Intervention  Services 


Randomized  Clinical  Trial 

—0  Completed 


I  Enrolled 


- GTest  =  1 


-  Attrition  =  1 


)  Control  =  2 


-  Attrition  =  1 


Completing  Consent  &  Baseline 


-4 

-  Attrition  =  4 


Pretest  -  Posttest  Comparison 

- Enrolled  =  3 


To  be  Enrolled  =  3 


Single-Subject  Design 

I -  0 
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a.  Randomized  Clinical  Trial.  We  have  randomly  assigned  12  children  to  the  technology- 
enhanced  training  group  and  waitlist-control  groups,  resulting  in  6  test-control  dyads.  The 
duration  of  in-home  services  (i.e.,  dose  of  ABA  therapy)  for  the  six  children  in  the 
technology-enhanced  training  group  are  depicted  below  in  Figure  4  below.  The  six 
children  are  depicted  on  the  y-axis  and  the  dose  of  in-home  services  (in  months)  is 
depicted  on  the  x-axis.  To  date,  four  test-control  dyads  (i.e.,  eight  participants)  have 
matriculated  through  the  randomized  clinical  trial  (participants  1  -  4),  with  the  training  for  a 
fifth  child  nearly  complete.  As  a  result,  we  have  completed  the  posttest  assessments  for 
the  four  children  in  the  test  group  as  well  as  the  posttests  for  their  peers  in  the  waitlist- 
control  group. 


Figure  4. 
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i.  Skills-based  Dependent  Measure:  The  dependent  measures  for  Experiment  3  include 
a  battery  of  standardized  assessments,  including,  the  Peabody  Picture  Vocabulary 
Test,  Expressive  Vocabulary  Test,  and  Mullen  Scale  of  Early  Learning.  In  addition  to 
these  standardized  assessments,  we  also  designed  an  assessment  that  sampled  skill 
domains  for  early  learners.  The  skills  that  were  assessed  was  informed  by  the 
domains  tested  in  the  Assessment  of  Basic  Language  and  Learning  Abilities. 

In  Figures  5  and  6  below,  the  results  for  each  matched  test-control  dyad  on  the  skills- 
based  dependent  measure  are  depicted.  The  different  skills  domains  are  depicted  on 
the  x-axis,  and  the  difference  score  between  the  pretest  and  posttest  is  depicted  on  the 
y-axis.  The  differences  scores  were  calculated  by  subtracting  the  percentage  of  trials 
with  correct  responding  on  the  posttest  from  the  percentage  of  trials  with  correct 
responding  on  pretest  for  each  domain.  Positive  scores  represent  an  improvement, 
and  negative  scores  represent  a  worsening.  Black  and  white  bars  denote  the  child 
assigned  to  the  test  and  control  groups,  respectively. 
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Figure  3. 


Figure  4. 
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For  the  Dyads  1  (top  panel)  and  2  (bottom  panel)  in  Figure  3,  the  children  assigned  to 
the  technology-enhanced  group  showed  improvement  across  the  majority  of  skill 
domains,  although  the  increase  in  performance  was  minimal  (20%  to  30%)  for  several 
domains.  Unexpectedly,  improvement  across  the  skill  domains  was  also  observed  for 
children  assigned  to  the  waitlist-control  group.  As  a  result,  increases  in  performance 
were  similar  across  children,  regardless  of  group  assignment.  By  contrast,  for  the  two 
dyads  in  Figure  4,  there  were  notable  skill  improvements  for  the  children  assigned  to 
the  technology-enhanced  group  compared  to  their  peer  in  the  control  group.  For  Dyad 
3,  the  performance  of  the  child  in  the  test  group  increased  by  >  20%  for  8  skills 
compared  to  2  skills  for  the  child  in  the  waitlist-control  group.  For  Dyad  4,  using  the 
same  analysis,  the  number  of  skills  with  improvement  was  1 1  for  the  child  in  the  test 
group  compared  to  4  for  the  child  in  the  waitlist-control  group.  As  a  result,  the 
technology-enhanced  early-intervention  services  was  associated  with  increased 
performance  in  skills  for  two  children  relative  to  two  peers  who  did  not  receive  the 
training.  The  outcomes  for  Dyads  3  and  4  are  examples  of  the  efficacy  results  we 
expect  in  the  randomized  clinical  trial  as  we  continue  to  increase  the  number  of  children 
enrolled. 


Standardized  Dependent  Measures:  The  results  for  the  Expressive  Vocabulary  Test 
(top  panel)  and  Peabody  Picture  Vocabulary  Test  (bottom  panel)  are  reported  in  Figure 
5.  The  four  test-control  dyads  are  depicted  on  the  x-axis,  and  the  children  in  the  test 
and  control  group  for  each  dyad  are  denoted  by  black  and  white  bars,  respectively. 

The  difference  scores  between  the  pretest  and  posttest  performances  are  reported  on 
the  y-axis,  and  these  scores  were  calculated  by  subtracting  the  raw  score  on  the 
posttest  from  the  raw  score  on  pretest  for  each  assessment.  We  did  not  observe 
consistent  differences  between  the  children  assigned  to  the  test  and  control  groups, 
although  the  sample  size  is  small  at  this  point. 


Figure  5. 
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The  results  for  the  Mullen  Scale  of  Early  Learning  are  reported  in  Figure  6.  The 
domains  in  the  assessment  are  denoted  on  the  x-axis,  and  one  of  the  four  test-control 
dyads  is  reported  in  each  panel.  The  difference  score  between  the  pretest  and  posttest 
is  denoted  on  the  y-axis  and  calculated  as  described  previously.  As  with  the  other 
standardized  assessments,  consistent  differences  between  the  children  assigned  to  the 
test  and  control  groups  were  not  observe,  although  the  sample  size  is  small  at  this 
point. 

Figure  6. 


Domain 
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b.  Pretest-Posttest  Evaluation.  Due  in  part  to  the  long  time  it  took  to  gain  consensual 

approval  of  the  human  protections  plan  from  the  DOD  and  the  UNMC  IRB,  we  are  behind 
the  proposed  schedule  for  recruiting,  enrolling,  and  completing  the  delivery  of  technology- 
enhanced  early  intervention  services  for  families  who  have  a  child  with  an  autism 
spectrum  disorder  (Experiment  3).  In  response,  after  communicating  with  Dr.  Stan  Niu, 
we  modified  the  research  methodology  to  help  to  address  our  recruitment  shortfall.  The 
research  modification  is  described  below. 

When  the  children  assigned  to  the  technology-enhanced  test  group  matriculate,  the  6 
months  of  early  intervention  services  for  their  peer  in  the  waitlist-control  group  will  start. 
We  are  providing  the  same  6-month  dose  of  early-intervention  services  to  the  children 
who  were  in  the  waitlist-control  group  and  including  their  data  in  the  statistical  analyses  of 
the  effects  of  our  technology-enhanced  services.  This  modification  will  enhance  the 
experimental  design  in  that  it  will  allow  for  both  an  across-group  comparison  between  the 
test  and  control  groups  as  well  as  a  within-subject  comparison  prior  to  and  after  the 
provision  of  our  technology-enhanced  early  intervention  services. 

The  duration  of  in-home  services  for  five  of  six  children  (attrition  for  one  family)  who  have 
completed  the  waitlist-control  group  are  depicted  below  in  Figure  7  below.  The  five 
children  are  depicted  on  the  y-axis  and  the  dose  of  in-home  services  (in  months)  is 
depicted  on  the  x-axis.  To  date,  three  children  have  started  to  receive  the  technology- 
enhanced  training  and  the  6-month  dose  of  services  for  one  child  has  been  completed. 
Data  analyses  will  be  conducted  and  reported  when  more  children  matriculate  through  the 
training. 

Figure  7. 
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c.  Continued  Recruitment  Efforts.  Over  the  last  year,  we  have  made  progress  in  obtaining 
outcomes  for  the  tutor-training  (Experiment  2)  and  parent-training  (Experiment  1)  aims. 

As  detailed  above,  the  preliminary  outcomes  strongly  support  the  efficacy  and  social 
acceptability  of  the  technology-enhanced  training  services  for  tutors  and  parents.  Given 
the  historical  difficulty  in  increasing  awareness  about  our  technology-enhanced  in-home 
services  to  families,  we  are  excited  by  our  progress  toward  designing  and  monitoring  early 
intervention  services  (10  children  who  have  or  will  be  receiving  services).  At  the  same 
time,  as  children  matriculate  through  the  training,  we  recognize  that  recruiting  additional 
test-control  family  dyads  is  paramount.  Below  we  list  the  following  developments  that  may 
increase  our  opportunities  for  recruitment: 

i.  Early  this  year,  we  spoke  with  Dr.  Ethan  Long,  who  is  the  executive  director  of  the 
Virginia  Institute  of  Autism.  He  is  leading  a  program  with  a  mission  to  serve  families 
with  a  child  with  autism  in  rural  Virginia  and  his  team  has  recently  been  in 
communication  with  personnel  at  Fort  Lee.  The  representatives  at  Fort  Lee  have 
indicated  that  there  is  large  need  for  parent  training  and  early  intervention  services 
for  local  military  families,  and  Dr.  Long  has  received  multiple  referrals.  Because  Dr. 
Long  and  his  team  cannot  serve  all  of  these  families,  he  has  agreed  to  refer  the 
families  to  us  for  potential  participation  in  our  grant. 

ii.  Tricare  has  established  an  Autism  Pilot  Program  that  expands  coverage  in  applied 
behavior  analysis  (ABA)  to  military  retirees;  prior  to  this  program,  only  active  military 
members  were  eligible  for  reimbursement  of  ABA  services.  The  pilot  program  is 
pertinent  to  the  current  grant  because  a  larger  number  of  families  are  now  eligible  to 
participate.  We  are  re-contacting  personnel  on  military  bases  to  describe  this 
opportunity  (e.g.,  exceptional  family  member  program  coordinators). 

iii.  Dr.  Luczynski  is  serving  as  the  program  chair  for  Division  25  (Behavior  Analysis)  in 
the  American  Psychology  Association.  One  week  ago  he  participated  in  a  leadership 
training  conference  in  Washington  D.C.  in  which  all  54  divisions  met  collectively. 
Division  19  is  the  Society  of  Military  Psychology,  and  Dr.  Luczynski  described  the 
aims  of  the  current  grant  to  the  program  chair  (Dr.  Nathan  Ainspan)  and  his 
disappoint  associated  with  the  challenge  of  effectively  distributing  information  about 
the  grant  to  military  families.  The  chair  of  Division  19  suggested  the  following 
courses  of  action: 

■  He  will  connect  Dr.  Luczynski  to  the  current  present  of  the  division  and  several 
past  presidents  who  have  been  strong  advocates  of  helping  military  families 
with  children  with  special  needs. 

■  He  suggested  that  we  contact  public  relations  agencies  who  have  a  strong 
interest  in  public  service  and  ask  the  agencies  if  they  would  be  willing  to  help 
us  distribute  the  information  regarding  the  grant  pro  bono.  He  said  that  this  is 
not  uncommon  when  the  cause  is  noteworthy  such  as  helping  military  families 
receive  the  high-quality  services  their  children  with  an  ASD  deserve. 

iv.  Toward  efforts  to  increase  our  recruitment  numbers  for  our  technology-enhanced  in- 
home  early  intervention  services  locally,  we  have  described  the  grant  in  a  newsletter 
that  is  published  by  the  Olson  Center  for  Women’s  Health,  which  reaches  over 
20,000  families  in  Nebraska  and  nearby  states. 
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V.  Pending  IRB  approval,  we  will  create  a  recorded  video  of  Dr.  Wayne  Fisher 

describing  the  mission  of  the  grant  and  the  impact  it  can  have  on  the  community  of 
military  families  with  a  child  with  autism.  We  would  also  ask  families  who  have 
received  early-intervention  services  from  our  center  to  describe  their  experiences. 
This  video  would  be  sent  to  news  stations  near  highly  populated  military  bases.  The 
video  may  help  with  recruitment  because  it  will  be  put  a  face  and  story  behind  who 
we  are  and  what  we  are  trying  to  accomplish. 

vi.  Two  weeks  ago,  we  had  a  meeting  with  the  Scott  Craven,  who  is  the  VP,  UHC  M&V 
(United  Healthcare  Military  &  Veterans)  for  Behavioral  Health  operations,  Shari 
Erickson,  who  is  the  VP,  UHC  M&V  for  Case  Management  (CM)  Operations  where 
the  autism  programs  reside,  Tracy  Sasso,  who  is  the  Associate  Director  for  Complex 
and  Specialized  CM,  and  Catherine  Mell,  who  is  the  Manager  of  Specialized  CM. 
During  this  meeting,  we  described  the  aims  of  the  grant  and  how  the  grant  may 
benefit  families  who  are  not  currently  receiving  applied-behavior-analysis  services. 
All  of  the  meeting  attendees  were  excited  about  the  program.  They  said  that  they 
are  going  to  determine  how  to  effectively  distribute  information  about  the  grant  to  the 
21  states  in  the  western  region  that  they  operate,  which  include  many  rural  areas 
with  military  families. 
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Key  Research  Accomplishments 

o  Technology-enhanced  Tutor-training  Curricula:  We  have  completed  the  preliminary 
statistical  analyses  for  the  randomized  clinical  trial  of  the  tutor-training  curricula  (4  test- 
control  dyads),  and  the  results  supported  the  efficacy  of  the  technology-enhanced 
teaching  procedures  for  improving  performance  on  both  primary  dependent  measures 
and  the  tutors  were  highly  satisfied  with  the  training  procedures,  content  covered,  and 
interactions  with  the  consultant. 

o  Technology-enhanced  Parent-training  Curricula:  Three  parents  in  the  technology- 
enhanced  group  and  five  parents  in  the  waitlist-control  group  have  matriculated  through 
the  evaluation.  As  observed  for  the  tutor-training  evaluation,  the  results  supported  the 
efficacy  of  the  web-based  teaching  procedures.  After  additional  participants  complete  the 
evaluation,  we  will  conduct  preliminary  statistical  analyses  and  submit  the  results  for 
publication. 

o  Technology-enhanced  Early-intervention  Services:  Four  test-control  dyads  of  children 
have  matriculated  through  the  evaluation.  Initial  results  provide  partial  support  for  the 
technology-enhanced  early-intervention  services.  The  completion  of  additional  dyads  is 
needed  prior  to  conducting  preliminary  analyses. 
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Reportable  Outcomes 

We  have  presented  the  preliminary  results  of  the  randomized  clinical  trial  for  the  tutor-training 
curriculum  at  our  national  conference  this  may,  the  Annual  Conference  of  the  Association  for 
Behavior  Analysis  International.  In  addition,  these  data  were  recently  published  in  published  in 
Research  in  Autism  Spectrum  Disorders  (5-year  Impact  Factor,  2.665)  in  their  latest  issue 
(Fisher,  Luczynski,  Flood,  Lesser,  Machado,  &  Pizza,  2014). 
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Conclusions 

Although  the  results  of  our  preliminary  studies  and  the  early  results  of  the  randomized  clinical 
trial  are  encouraging,  it  is  premature  to  draw  any  firm  conclusions  at  this  juncture  of  the  award. 


24 


References 

Fisher,  W.  W.,  Luczynski,  K.  C.,  Hood,  S.  A,  Lesser,  A.  D,  Machado,  M.  A.,  &  Piazza,  C.  C. 

(2014).  Preliminary  findings  of  a  randomized  clinical  trial  of  a  virtual  training  program  for 
applied  behavior  analysis  technicians.  Research  in  Autism  Spectrum  Disorders,  8,  1044- 
1054. 

Luczynski,  K.  C.,  Fisher,  W.  W.,  Hood,  S.  A.,  Lesser,  A.  D.,  Machado,  M.  A.  (2014,  May). 

Preliminary  findings  of  a  randomized  clinical  trial  of  a  virtual  training  program  for  applied 
behavior  analysis  technicians.  Paper  to  be  presented  at  the  40*^  Annual  Convention  for 
the  Association  for  Behavior  Analysis  International,  Chicago,  IL. 


25 


Appendix  1 


See  appended  copy  below  of  the  published  manuscript. 


Research  in  Autism  Spectrum  Disorders  8  (2014)  1044-1054 


ELSEVIER 


Contents  lists  available  at  SclenceDirect 

Research  in  Autism  Spectrum  Disorders 

Journal  homepage:  http://ees.elsevler.com/RASD/default.asp 


Research  in 
Autism  Spectrum 
Disorders 


Preliminary  findings  of  a  randomized  clinical  trial  of  a  virtual 
training  program  for  applied  behavior  analysis  technicians'^ 


CrossMark 


Wayne  W.  Fisher*,  Kevin  C.  Luczynski,  Stephanie  A.  Hood,  Aaron  D.  Lesser, 
Mychal  A.  Machado,  Cathleen  C.  Piazza 

The  University  of  Nebraska  Medical  Center’s  Monroe-Meyer  Institute,  United  States 


ARTICLE  INFO 


ABSTRACT 


Article  history: 

Received  25  February  2014 
Accepted  3  May  2014 
Available  online  18  June  2014 


Keywords: 

Applied  behavior  analysis 
Autism 

Behavioral  skills  training 

Early  intensive  behavioral  interventions 

Telehealth 

Virtual  care 


As  the  demand  for  applied  behavior  analysis  (ABA)  services  for  children  with  an  autism 
spectrum  disorder  continues  to  grow,  it  is  critical  to  develop  efficient,  effective,  and  widely 
accessible  procedures  for  training  technicians  to  implement  ABA  interventions.  One 
approach  would  be  to  develop  efficacious  training  programs  that  could  be  delivered  over 
the  Internet  via  a  virtual  private  network  (VPN).  In  the  current  study,  we  developed  a  40-h 
virtual  training  program  in  which  participants  completed  e-learning  modules  and  also 
received  behavioral  skills  training  over  a  VPN  to  implement  behavior  reduction  and  skill 
acquisition  protocols  in  both  discrete-trail  and  play-based  formats.  This  virtual  training 
program  was  evaluated  in  a  randomized-clinical  trial  (RCT)  using  direct-observation 
measures  on  the  implementation  of  discrete-trial  training  and  play-based  procedures  as 
the  primary  dependent  variables  (which  were  also  collected  via  a  VPN).  Participants  in  the 
treatment  group  showed  robust  and  statistically  significant  improvement  in  their 
implementation  of  behavior  reduction  and  acquisition  programs  under  both  discrete-trial 
and  play-based  formats,  and  they  rated  the  training  as  highly  socially  acceptable.  These 
preliminary  results  from  an  ongoing  RCT  suggest  that  this  effective,  convenient,  and 
socially  acceptable  virtual  training  program  has  the  potential  to  extend  access  to  ABA 
services  to  families  in  rural  and  other  underserved  areas  or  populations. 
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1.  Introduction 

Autism  spectrum  disorder  (ASD)  is  a  disorder  of  the  brain  that  involves  impairments  in  social  relatedness  and  language 
and  inflexible  or  repetitive  behaviors.  Prevalence  estimates  have  grown  more  than  20  fold  since  the  1980s,  and  ASD  currently 
affect  about  1  in  88  children  in  the  United  States  (Burd,  Fisher,  &  Kerbeshan,  1987;  Centers  for  Disease  Control  and 
Prevention,  2012).  Moreover,  a  recent  study  in  South  Korea  used  more  aggressive  case-ascertainment  methods  than  most 
prior  studies  and  reported  a  prevalence  of  2.6%  or  about  1  in  38  children  (Kim  et  al.,  201 1 ).  These  increases  in  the  number  of 
diagnosed  cases  of  ASD  have  greatly  expanded  the  demand  for  intervention  services  (Dave  &  Fernandez,  2012).  This 
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increased  demand  is  also  due,  in  part,  to  the  long-term  outcomes  that  have  been  reported  to  typically  occur  for  children  with 
ASD  in  the  absence  of  effective  treatment. 

Without  intensive  and  appropriate  treatments,  the  long-term  outcomes  for  children  with  ASD  remain  grim.  In  a  long¬ 
term  follow-up  study  of  adults  affected  by  an  ASD,  only  26%  had  friends,  13%  had  independent  jobs  (mostly  low  paying 
positions),  and  4%  lived  on  their  own  (Howlin,  2005;  Howlin,  Goode,  Hutton,  &  Rutter,  2004).  It  is  fairly  common  for 
individuals  with  ASD  to  live  with  and  be  dependent  upon  parents,  siblings,  or  both  throughout  their  adult  life.  The  burden 
and  stressors  placed  on  families  with  a  child  with  an  ASD  are  substantial.  The  divorce  rate  among  parents  of  children  with  an 
ASD  has  been  estimated  to  be  above  80%,  and  immediate  family  members  are  at  increased  risk  for  the  development  of  stress- 
related  mental  health  disorders  (Dumas,  Wolf,  Fisman,  &  Culligan,  1991;  Feldman  et  al.,  2007;  Lofholm,  2008).  The  rapid 
growth  in  the  prevalence  of  ASD,  combined  with  the  deleterious  effects  the  disorder  has  on  affected  children  and  their 
families,  has  to  led  to  a  spike  in  demand  for  effective  autism  services,  and,  in  particular,  for  services  with  strong  empirical 
evidence  supporting  treatment  efficacy  (National  Autism  Center,  2009). 

The  treatment  for  ASD  that  has  the  most  empirical  support  is  called  applied  behavior  analysis  (ABA;  Eikeseth,  2009; 
Eldevik  et  al.,  2009;  Howlin,  Magiati,  8;  Charman,  2009).  Early  intensive  behavioral  intervention  (EIBl)  is  a  specific  approach 
to  the  treatment  of  young  children  with  ASD  that  applies  the  basic  tenets  of  ABA  (Baer,  Wolf,  &  Risley,  1968).  ABA  is  based,  in 
part,  on  the  principle  of  “selection  by  consequences”  in  which  responses  that  produce  favorable  outcomes  become  more 
probable  and  ones  that  produce  unfavorable  outcomes  become  less  probable.  In  EIBl  services,  the  procedures  from  the  field 
of  ABA  are  used  to  (a)  reduce  the  symptoms  and  problem  behaviors  associated  with  ASD  and  (b)  replace  those  problem 
behaviors  with  appropriate  skills  in  language,  socialization,  play,  cognition,  academics,  motor  development,  and  daily  living 
skills.  Thus,  in  an  EIBl  program,  events  in  the  environment  are  re-arranged  such  that  appropriate  behaviors  are  learned  and 
re-occur  and  problem  behaviors  are  decreased  or  eliminated. 

A  substantial  and  growing  number  of  investigations  have  found  that  EIBl  produced  marked  gains  in  IQ(average  increases 
of  about  20  IQ  points;  Rogers  &  Vismara,  2008).  In  addition,  a  number  of  empirical  studies  have  found  that  ABA-based 
programs  are  more  effective  than  eclectic  therapies  (e.g.,  Howard,  Sparkman,  Cohen,  Green,  &  Stanislaw,  2005).  For  example, 
Howard  et  al.  compared  the  effectiveness  of  EIBl  to  an  “eclectic”  form  of  therapy,  which  consisted  of  a  variety  of  therapy 
methods  (e.g.,  sensory  integration  therapy,  picture-exchange  communication,  and  general  preschool  activities).  Both  groups 
received  a  minimum  of  25  h  of  therapy  per  week.  Program  effectiveness  was  evaluated  by  comparing  child  outcomes  on  a 
variety  of  measures,  including  intellectual  functioning,  non-verbal  problem  solving,  language,  and  adaptive  skills.  Across 
each  measure,  children  who  received  EIBl  scored  higher  than  those  who  received  the  eclectic  form  of  therapy.  Moreover, 
these  gains  were  evident  during  follow-up  evaluations  conducted  at  an  average  of  14  months  after  conclusion  of  the  study. 
These  outcomes,  combined  with  those  of  other  evaluations,  suggest  that  EIBl  models  represent  the  most  effective  form  of 
therapy  for  children  with  ASD  (Rogers  &  Vismara,  2008).  In  a  report  by  the  American  Academy  of  Pediatrics  (Myers  & 
Johnson,  2007),  they  noted  that,  “The  effectiveness  of  ABA-based  intervention  in  ASDs  has  been  well  documented. . (p. 
1164).  They  also  noted  that  EIBl  and  functional  analysis  were  two  of  the  approaches  to  the  treatment  of  ASD  that  had  the 
most  empirical  support,  and  that  a  minimum  of  25  h  per  week  of  EIBl  and  a  low  child-to-therapist  ratio  (i.e.,  1:1)  were 
important  variables  affecting  the  success  of  the  treatment  programs. 

Most  EIBl  services  are  delivered  using  a  tiered  model  in  which  the  professional  behavior  analyst  designs  and  supervises 
treatment  protocols  that  are  delivered  by  a  behavioral  technician  (Behavior  Analyst  Certification  Board,  Inc.  [BACB],  2012). 
This  tiered  model  facilitates  the  cost-effective  delivery  of  EIBl  services  at  effective  intensity  levels  (e.g.,  25  or  more  hour  per 
week)  that  have  strong  empirical  support  (Myers  &  Johnson,  2007),  because  the  professional  behavior  analyst  can  manage 
multiple  cases  and  most  of  the  direct  service  is  provided  at  a  lower  cost  by  the  technician  (BACB).  However,  few  if  any  criteria 
or  guidelines  are  available  for  determining  what  training  and  skills  technicians  should  receive  prior  to  working  with  children 
with  ASD,  although  the  BACB  has  announced  plans  to  offer  a  new  credential  for  these  direct-line  staff  called  the  “Registered 
Behavioral  Technician.” 

Rispoli,  Neely,  Lang,  and  Ganz  (2011)  recently  reviewed  the  extant  literature  on  training  technicians  to  implement 
interventions  for  persons  with  ASD  and  found  only  12  experimentally  sound  studies.  These  studies  evaluated  the  effects  of 
technician  or  paraprofessional  training  using  single-case  designs  and  involved  a  total  of  39  participants.  Nine  of  the  studies 
were  conducted  in  schools,  and  none  of  the  studies  involved  technicians  who  were  implementing  EIBl  services  in  the  homes 
of  children  with  ASD,  where  there  is  typically  infrequent  on-site  supervision  from  a  professional.  All  of  the  studies  involved 
training  the  participants  on  a  specific  procedure  (e.g.,  discrete-trial  training),  despite  the  fact  that  technicians  are  typically 
expected  to  implement  multiple  types  of  behavior-reduction  and  skill-acquisition  protocols.  Finally,  none  of  the  studies 
evaluated  the  effects  of  technician  training  using  an  RCT.  These  findings  suggest  that  additional  research  is  needed  on 
technician  training  that  (a)  promotes  more  convenient  and  cost-effective  training  methods,  (b)  expands  the  number  and 
variety  of  skills  taught  to  technicians,  (c)  assesses  procedures  for  training  technicians  to  implement  home-based  ABA 
programs,  and  (d)  evaluates  the  effects  of  technician  training  using  an  RCT. 

Heitzman-Powell,  Buzhardt,  Rusinko,  and  Miller  (2014)  recently  reported  on  a  virtual,  e-learning  program  for  training 
ABA  principles  and  procedures  to  parents  of  children  with  ASD,  thus  providing  preliminary  evidence  on  a  potentially  cost- 
effective  training  method.  In  their  initial  study,  seven  parents  completed  the  Online  and  Applied  System  for  Intervention 
Skills  (OASIS)  program  within  a  pretest-posttest  design  (Buzhardt  &  Heitzman-Powell,  2005).  The  OASIS  program  consists  of 
eight  modules  covering  topics  such  as  an  introduction  to  behavioral  treatment,  defining  and  measuring  behavior,  skill 
assessment  and  intervention,  and  stimulus  control.  During  the  first  section  of  each  module,  the  parent  completed  an  online. 
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interactive  tutorial  designed  to  improve  knowledge  on  the  target  topic.  Parents  were  then  required  to  score  90%  or  better  on 
a  20-item  test  prior  to  beginning  the  second  section  of  the  module,  which  consisted  of  the  parent  practicing  the  skills 
targeted  in  the  module  with  their  child  while  receiving  coaching  and  feedback  from  a  professional  via  a  telehealth 
connection.  Other  quasi-experimental  studies  have  used  similar  methods  to  teach  knowledge  of  behavioral  intervention 
procedures  or  to  train  therapists  to  implement  the  Early  Start  Denver  Model  (Hamad,  Serna,  Morrison,  &  Fleming,  2010; 
Vismara,  Young,  Stahmer,  Griffith,  &  Rodgers,  2009).  We  have  found  only  one  study  in  which  an  RCT  was  used  to  evaluate 
virtual  training  in  the  principles  and  procedures  of  ABA,  but  the  dependent  variable  for  that  study  was  knowledge  of  ABA  as 
measured  on  a  multiple-choice  test  Qang,  Dixon,  Tarbox,  Granpeesheh,  Kornack,  &  de  Nocker,  2012).  Thus,  the  extent  to 
which  the  participants  could  accurately  implement  a  variety  of  ABA  procedures  remains  unknown.  Therefore,  the  purpose  of 
the  current  investigation  was  to  extend  these  prior  studies  on  virtual  training  by  using  an  RCT  to  systematically  evaluate  the 
effectiveness  of  a  more  comprehensive  virtual  program  for  training  technicians  to  both  understand  and  implement  a  variety 
of  ABA  principles  and  procedures  used  in  EIBI  services  for  young  children  with  ASD. 

2.  Method 

2.3.  Participants  and  settings 

Eight  participants  (seven  females,  one  male)  ranging  in  age  from  21  to  over  50  years  old  were  recruited  by  the  Munroe- 
Meyer  Institute’s  Virtual  Care  Program  at  the  University  of  Nebraska  Medical  Center  (UNMC)  as  part  of  an  ongoing  multi¬ 
study  project  aimed  at  demonstrating  that  virtual  technologies  and  skills-training  strategies  can  be  used  to  expand  and 
enhance  the  availability  of  EIBI  services  to  children  with  an  ASD.  The  participants  (henceforth  called  technicians  to 
distinguish  them  from  the  confederates  described  subsequently)  were  recruited  to  provide  direct  services  (after  training  was 
complete)  to  active  military  families  with  children  with  an  ASD  through  the  Autism  Demonstration  Project  (Defense  Health 
Agency,  2008).  Therefore,  we  called  the  Exceptional  Family  Member  Program-Coordinators  (EFMP-C)  at  all  military  bases 
throughout  the  United  States  (i.e..  Army,  Air  Force,  Marine  Corps,  and  Navy)  to  explain  the  purpose  of  the  study.  We  recruited 
technicians  via  flyers  that  were  sent  to  the  EFMP-Cs  at  each  base.  In  addition,  we  contacted  all  directors  of  University  Centers 
for  Excellence  in  Developmental  Disabilities  and  sent  them  recruitment  flyers  to  distribute.  We  also  recruited  behavioral 
technicians  that  were  hired  through  the  Center  for  Autism  Spectrum  Disorders  Clinic  at  UNMC’s  Munroe-Meyer  Institute  to 
provide  in-home,  EIBI  services. 

We  enrolled  technicians  if  they  (a)  were  at  least  19  years  of  age;  (b)  completed,  or  were  enrolled  in,  an  associate’s  or 
bachelor’s  degree  program  in  psychology  or  a  related  field;  (c)  completed  at  least  12  semester  credit  hours  at  an  accredited 
college  or  university;  and  (d)  had  no  prior  training  in  implementing  ABA  interventions.  Technicians  were  excluded  if  they  did 
not  meet  the  aforementioned  inclusion  criteria  or  could  not  complete  the  physical  requirements  of  the  tasks  involved  in  ABA 
therapy,  which  require  therapists  to  be  able  to  speak,  hear,  see,  sit,  stand,  walk,  and  lift  40  lbs.  All  procedures  were  approved 
by  the  Institutional  Review  Boards  at  UNMC  and  the  Autism  Research  Program  at  the  Congressionally  Directed  Medical 
Research  Program  within  the  Department  of  Defense.  Each  technician  provided  written  consent  to  participate  in  the  study. 
Technicians  completed  the  training  in  their  home  or  at  a  nearby  location  where  they  had  access  to  a  high-bandwidth  Internet 
connection  (e.g.,  a  study  room  in  a  college  library). 

3.  Dependent  measures 

The  two  primary  dependent  measures  were  the  Behavioral  Implementation  of  Skills  for  Work  Activities  (BISWA)  and  the 
Behavior  Implementation  of  Skills  for  Play  Activities  (BISPA).  The  BISWA  and  BISPA  are  direct-observation  measures  designed  to 
assess  the  technician’s  skill  at  implementing  ABA  procedures  in  a  discrete-trial  format  and  a  play-based  format,  respectively. 
The  technician,  whose  skills  were  assessed,  was  at  one  location  (e.g.,  their  home)  along  with  a  friend  or  colleague  who  was 
recruited  to  play  the  role  of  a  child  with  an  ASD  during  the  assessment  (henceforth  called  the  confederate;  written  consent  to 
participate  was  required).  The  researcher  verbally  directed  the  responses  of  the  confederate  from  another  location  (e.g.,  an 
office  at  UNMC)  via  a  virtual  private  network  (VPN ;  i.e.,  a  password  protected  and  encrypted  connection  between  a  computer 
at  the  technician’s  location  and  a  computer  at  UNMC).  We  mailed  the  technician  a  bluetooth  earpiece,  webcam,  and 
materials  needed  for  conducting  the  BISWA  and  BISPA,  and,  if  needed,  a  laptop  computer.  The  confederate  wore  the 
bluetooth  earpiece  that  was  connected  to  the  VPN  on  the  computer  or  phone,  which  allowed  the  researcher  to  give  verbal 
instructions  to  the  confederate  without  the  technician  hearing  the  instructions  (through  the  earpiece)  or  to  both  the 
technician  and  the  confederate  (through  the  computer  speaker). 

The  BISWA  assessment  consisted  of  36  trials  and  the  BISPA  assessment  consisted  of  20  trials.  The  assessments  were  video 
recorded  for  subsequent  scoring.  At  the  start  of  an  assessment  session,  the  technician  had  a  set  of  stimulus  cards  (previously 
sent  through  the  mail)  that  were  used  to  teach  the  confederate  receptive-identification  skills  (e.g.,  pointing  to  the  picture  of  a 
dog  after  the  technician  said  “Touch  dog.’’).  On  each  trial,  the  researcher  instructed  the  confederate  (via  the  bluetooth 
earpiece)  to  respond  to  the  technician  in  a  number  of  different  ways  representative  of  the  range  of  possible  responses  a 
technician  would  typically  encounter  when  working  with  a  child  with  an  ASD.  Over  the  course  the  assessments,  the 
confederate  responded  with  an  independent  correct  response  and  no  problem  behavior  following  the  initial  instruction  on 
four  trials  that  were  interspersed  in  a  quasi-random  fashion  over  the  course  of  the  assessment.  On  an  additional  16  trials 
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following  the  initial  instruction,  the  researcher  directed  the  confederate  to  challenge  the  technician  by  displaying  (a)  no 
response,  (b)  an  incorrect  response,  (c)  stereotypy,  or  (d)  other  problem  behavior  (aggression,  disruption,  echolalia,  and 
negative  vocalizations).  Within  these  same  1 6  trials,  we  further  challenged  the  technician  by  having  the  confederate  display 
one  of  the  following  responses  after  the  confederate  made  an  error:  (a)  a  prompted  correct  response,  (b)  a  prompted 
incorrect  response,  (c)  no  response,  or  (d)  other  problem  behavior. 

After  the  session,  trained  observers  scored  the  video  recording,  which  was  recorded  via  the  computer,  of  the  technician’s 
responses  in  each  BISWA  assessment  for  the  occurrence  of  correct  and  incorrect  implementation  of  behavioral  procedures 
(e.g.,  gains  child’s  attention,  provides  clear  instruction,  provides  descriptive  praise,  provides  tangible  or  edible 
reinforcement,  and  ignores  problem  behavior).  The  top  half  of  Table  1  shows  the  programmed  confederate  responses, 
and  the  bottom  half  shows  the  target  responses  that  were  scored  for  the  technician. 

The  BISPA  was  similar  to  the  BISWA  except  that  the  technician  skills  were  assessed  during  a  play-based,  naturalistic 
training  context  in  which  the  technician  was  instructed  to  increase  the  confederate’s  appropriate  behavior  and  decrease 
problem  behavior.  As  with  the  BISWA,  at  prescribed  times,  the  researcher  instructed  the  confederate  to  display  various 
responses.  For  10  of  these  proscribed  occasions,  interspersed  over  the  course  of  the  session  in  a  quasi-random  fashion,  the 
confederate  emitted  each  of  the  following  response  twice:  (a)  engaged  in  a  bid  for  joint  attention,  (b)  initiated  conversation, 
(c)  engaged  in  functional  play,  (d)  described  an  emotion  displayed  by  the  technician,  and  (e)  initiated  play  with  the 
technician.  On  the  remaining  10  occasions,  the  researcher  directed  the  confederate  to  engage  in  each  of  the  following 
problem  behaviors  twice:  (a)  aggression,  (b)  disruption,  (c)  echolalia,  (d)  negative  vocalizations,  and  (e)  stereotypy.  Trained 
observers  subsequently  scored  the  video  recording  of  the  technician’s  correct  and  incorrect  responses  to  these  confederate 
behaviors.  That  is,  the  observers  scored  whether  the  technician  responded  to  appropriate  confederate  behaviors  with 
descriptive  praise  and  reinforcement  and  to  the  confederate’s  problem  behavior  with  extinction.  The  top  half  of  Table  2 
shows  the  programmed  confederate  responses,  and  the  bottom  half  shows  the  target  responses  that  were  scored  for  the 
technician. 

Although  the  BISPA  did  not  consist  of  discrete  trials  initiated  by  the  technician,  we  divided  each  session  into  a  series  of 
trials  demarcated  by  the  occurrence  of  a  programmed  confederate  response  for  scoring  purposes.  For  example,  if  the  first 
programmed  response  was  for  the  confederate  to  display  aggression  20  s  into  the  session,  the  first  trial  started  when  that 
aggressive  response  was  emitted  by  the  confederate  and  continued  until  the  next  programmed  confederate  response 
occurred.  Observers  scored  whether  the  technician  responded  correctly  or  incorrectly  to  each  of  the  confederate’s 
programmed  responses.  For  example,  if  the  programmed  confederate  response  was  a  negative  vocalization  (e.g.,  a  scream), 
the  correct  technician  response  was  to  ignore  the  scream  and  continue  with  the  ongoing  activity.  If  the  programmed 
confederate  response  was  functional  play  (e.g.,  rolling  a  car),  the  correct  technician  response  was  to  provide  descriptive 
praise  and  reinforcement  (e.g.,  "That  was  super!  You  rolled  the  car  a  long  way!’’,  and  then  immediately  roll  the  car  back  to 
the  confederate). 

Table  1 

Behavior  Implementation  of  Skills  for  Work  Activities  (BISWA). 


Dependent  measures 

Programmed  confederate  responses 
Independent  correct 
Prompted  correct 
Incorrect 

Prompted  incorrect 

No  response 

Stereotypy 

Problem  behavior 

Aggression 

Disruption 

Echolalia 

Negative  vocalization 
Technician  responses 
Instructions 
Behavior  specific 
Clear 

Phrased  as  a  demand 

Gain  the  child’s  attention 
Following  a  correct  response 
Descriptive  praise 

Reinforcement  delivery 
Following  the  first  error 
Model  prompt 
Following  the  second  error 
Physical  prompt 


Operational  definition 


Touching  the  target  stimulus  card  within  5  s  of  the  instruction 
Touching  the  target  stimulus  card  within  5  s  of  the  model  prompt 
Touching  a  nontarget  stimulus  card  stimulus  within  5  s  of  the  instruction 
Following  a  model  prompt,  touching  a  nontarget  stimulus 

Not  touching  a  stimulus  or  engaging  in  problem  behavior  within  5  s  of  the  instruction 
Repetitive  body  rocking  or  contorting  hand  in  front  of  face 

Hitting  the  technician  with  body  or  objects 
Banging  surfaces,  throwing  or  swiping  of  instructional  materials 
Repeating  one  or  several  words  said  by  the  technician 
Crying  or  screaming  at  a  volume  above  conversational  level 


The  action  and  target  stimulus  is  stated  (example,  “Touch  dog;’’  nonexample,  “Show  me  what  you  know”) 
Only  the  action  and  target  stimulus  is  stated  (example,  “Touch  dog;”  nonexample,  “That  is  a  pretty  dog, 
touch  dog”) 

The  action  and  target  stimulus  is  stated  as  a  direct  imperative  and  not  as  a  question  or  declarative 
(example,  “Touch  dog;”  nonexamples,  “Will  you  touch  dog?”  and  “It’s  time  to  touch  dog”) 

Prompt  the  confederate  to  look  at  the  technician  or  the  stimulus  cards  prior  to  the  instruction 

Vocal  praise  in  which  the  action  and  target  stimulus  is  stated  (example,  “Super  smart  touching  dog;” 
nonexample,  “Nice  work”) 

Delivery  of  a  high  preferred  item  (i.e.,  an  edible  or  tangible)  within  5  s  of  the  correct  response 
Touch  the  target  stimulus 

Hand-over-hand  physical  guidance  to  the  target  stimulus 
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Table  2 

Behavior  Implementation  of  Skills  for  Play  Activities  (BISPA). 


Dependent  measures 

Programmed  confederate  responses 
Appropriate  behavior 
Joint  attention 
Initiates  conversation 

Functional  play 

Describes  others  emotions 

Initiates  play  with  technician 

Problem  behavior 

Aggression 

Disruption 

Echolalia 

Negative  vocalizations 
Stereotypy 
Technician  responses 
Descriptive  praise 
Reinforcement  delivery 
Extinction 


Operational  definition 


Any  vocal  or  gestural  response  to  direct  the  technician’s  attention  toward  an  object  or  activity 
Appropriate  spoken  verbalization  that  is  not  in  direct  response  to  a  technician’s  question  and  does  not 
involve  requests  for  tangible  or  edibles  items 

Independent  use  of  play  materials  in  a  manner  consistent  with  their  intended  purpose 
Appropriate  vocalizations  that  describes,  asks  about,  or  shows  empathy  for  the  emotions  of  the  technician 
Any  vocal  or  gestural  response  to  begin  a  play  activity  with  the  technician 

Hitting  the  technician  with  body  or  objects 
Banging  surfaces,  or  throwing,  swiping  instructional  materials 
Repetition  of  words  or  phrases  said  by  the  technician 
Crying  or  screaming  at  a  volume  above  conversational  level 
Repetitive  body  rocking  or  contorting  hand  in  front  of  face 

Vocal  praise  in  which  the  skill  emitted  by  confederate  is  stated 

Delivery  of  a  high  preferred  item  (i.e.,  an  edible  or  tangible)  within  5  s  of  the  correct  response 
Absence  of  a  stimulus  change  following  problem  behavior 


4.  Interobserver  agreement,  observer  blinding,  and  procedural  integrity 

Interobserver  agreement  was  calculated  by  having  two  independent  observers  simultaneously  and  independently  collect 
data  on  technicians’  responses  for  a  percentage  of  the  BISWA  and  BISPA  assessments  and  then  comparing  the  observers’ 
records.  During  the  pretest,  all  observers  were  blind  to  group  membership  because  technicians  were  randomly  assigned  to 
the  treatment  or  control  groups  after  the  pretest  was  completed.  For  50%  of  the  posttests,  a  trained  observer  who  was  blind  to 
group  membership  and  the  study’s  hypotheses  scored  technician  responses  from  the  video  recordings  of  the  BISWA  and 
BISPA.  For  purposes  of  calculating  interobserver  agreement,  each  assessment  was  divided  into  a  series  of  trials  as  described 
above,  and  the  total  number  of  trials  on  which  the  two  observers  agreed  was  divided  by  the  total  number  of  trials  scored  and 
the  resulting  quotient  was  converted  to  a  percentage.  An  agreement  was  scored  if  both  observers  recorded  the  same 
technician  skill  as  having  occurred  or  not  occurred  during  each  trial. 

In  both  the  groups,  interobserver  agreement  was  scored  for  56%  of  the  pretest  and  posttest  sessions.  Mean  agreement 
levels  across  participants  were  86%  and  83%  for  the  BISWA  and  BISPA,  respectively.  Mean  agreement  levels  between  the 
observer  who  was  blind  to  group  membership  and  study  hypotheses  and  the  unblinded  observer  were  85%  and  86%  for  the 
BISWA  and  BISPA,  respectively.  Thus,  no  evidence  of  observer  bias  was  discovered. 

Procedural  integrity  on  the  confederates’  implementation  of  the  role-play  activities  was  scored  for  1 00%  of  the  pretest  and 
posttest  sessions.  Mean  procedural  integrity  levels  across  confederates  were  94.4%  (range,  81-100%)  and  99.8%  (range, 
95-100%)  for  the  BISWA  and  BISPA,  respectively.  These  data  indicate  that  the  confederates  completed  the  prescribed 
responses  with  high  levels  of  accuracy,  which  was  expected  because  researchers  provided  real-time  guidance. 

5.  Procedure 

Training  for  technicians  included:  (a)  completing  e-learning  modules  (described  subsequently)  at  times  convenient  to 
their  schedule  and  (b)  receiving  real-time  direct  instruction  with  descriptive  feedback  from  researchers.  Regarding  the  latter, 
technicians  observed  researchers  modeling  the  target  skills,  and  the  technicians  behaviorally  rehearsed  those  procedures 
with  coaching  and  feedback  from  researchers  via  a  video-conference-meeting  platform  (i.e.,  GoToMeeting).  Each  technician 
was  provided  with  a  webcam  (Logitech  C525  or  C615),  bluetooth  headset  (Plantronics  Voyager  Pro),  ethernet  connection 
cords,  and  the  materials  needed  for  the  training.  During  the  study,  technicians  were  free  to  proceed  through  the  e-learning 
modules  at  their  own  pace;  however,  the  researchers  encouraged  the  technicians  to  set  a  goal  each  week  and  provided 
positive  feedback  for  meeting  the  goal. 

5.3.  Pretest 

Prior  to  random  assignment  to  either  the  treatment  or  control  groups,  technicians  completed  an  initial  assessment  to 
determine  their  baseline  skill  levels  using  the  BISWA  and  BISPA  assessments  as  described  above. 

5.2.  Randomization 

After  technicians  completed  the  BISWA  and  BISPA  assessments,  they  were  matched  into  pairs  (based  on  how  closely  in 
time  they  completed  the  pretests)  and  then  randomly  assigned  to  either  the  treatment  or  control  group.  Technicians 
assigned  to  the  treatment  group  completed  the  virtual  training  program  immediately;  technicians  in  the  control  group  were 
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placed  on  a  waitlist  and  did  not  receive  training  until  their  matched  partner  completed  the  training  program  and  both 
technicians  completed  a  second  set  of  BISWA  and  BISPA  assessments  (i.e.,  posttest). 

5.3.  Virtual,  e-learning  program 

The  e-learning  training  program  consisted  of  17  multi-media  modules,  each  lasting  40-60  min,  and  was  divided  into 
three  sections.  The  content  of  the  modules  was  designed  to  provide  an  overview  of  a  behavior-analytic  principles  and 
strategies  related  to  applying  ABA  to  child  behavior  as  well  as  demonstrations  of  training  strategies  through  pictures,  video, 
and  narration  for  each  slide.  Technicians  accessed  each  module  from  UNMC’s  Blackboard  website  and  completed  the 
modules  in  a  sequential  fashion.  Blackboard  is  a  content  delivery  service  commonly  used  by  universities.  The  modules  in 
Section  1  included:  (a)  Ethics,  Least  Restrictive  Environment,  and  Vulnerable  Populations;  (b)  Extinction;  (c)  Punishment:  (d) 
Preference  Assessments  and  Positive  Reinforcement:  (e)  Stimulus  Control  and  Motivating  Operations:  and  (f)  Stimulus 
Generalization  and  Maintenance.  The  modules  in  Section  2  included:  (a)  Introduction  to  Verbal  Behavior,  (b)  Discrete-Trial 
Training,  (c)  Matching-to-Sample  and  Discrimination  Training,  (d)  Response  Prompting,  and  (e)  Response  Chaining.  The 
modules  in  Section  3  included:  (a)  Natural  Environmental  Training,  (b)  Assessment  and  Treatment  of  Problem  Behavior,  (c) 
Compliance,  (d)  Preventing  Problem  Behavior,  (e)  Putting  it  Together  I-Integrating  Structured  Training  Strategies,  and  (f) 
Putting  it  Together  11-lntegrating  Play-Based  Training  Strategies.  A  10-item,  multiple-choice  quiz  was  created  for  each 
module  and  technicians  had  to  correctly  answer  80%  or  more  of  the  questions  before  proceeding  to  the  subsequent  module. 

5.4.  Scripted  role-plays 

Role-plays  were  programmed  for  six  of  the  modules  and  were  designed  to  provide  technicians  with  opportunities  to 
practice  the  skills  that  were  covered  in  the  module.  For  the  role-plays,  the  researcher,  technician,  and  a  confederate 
established  a  virtual  meeting  and  bluetooth  audio  connection  as  described  above  for  the  pretests  and  posttests.  At  the 
beginning  of  the  role-play,  technicians  were  instructed  to  show  the  researcher  how  to  implement  the  target  skill  based  on 
the  instructions  and  visual  exemplars  provided  in  the  module.  For  example,  for  one  situation  during  the  role-play  for  the 
Response  Prompting  module,  technicians  were  asked  to  implement  a  full-physical  prompt  with  a  5-s  prompt  delay.  Each 
role-play  consisted  of  several  situations  (or  multiple  exemplars;  Sprague  &  Horner,  1984;  Stokes  &  Baer,  1977;  Stokes  & 
Osnes,  1 989)  for  which  the  researcher  instructed  the  confederate  (via  the  earpiece  so  that  the  technician  would  not  hear  the 
instruction)  to  emit  the  child  responses  scripted  for  each  situation  (i.e.,  trial).  Each  confederate  response  was  programmed 
multiple  times  to  obtain  repeated  opportunities  for  the  technician  to  practice  the  response  and  receive  feedback.  If  the  skill 
was  implemented  incorrectly,  the  researcher  provided  descriptive  feedback  by  detailing  the  correct  response  and,  on 
occasion,  by  modeling  correct  implementation  of  the  skill  via  the  live-video  connection;  next,  another  opportunity  to 
practice  was  arranged.  For  the  final  two  role-play  modules  (Putting  it  Together  I-lntegrating  Structured  Training  Strategies 
and  Putting  it  Together  Il-lntegrating  Play-Based  Training  Strategies),  role-play  training  continued  until  the  technician 
demonstrated  independent  mastery  of  the  skills  at  80%  or  above. 

5.5.  Posttest 

The  posttest  was  identical  to  the  pretest  except  that  it  occurred  after  the  treatment  group  completed  the  virtual  training 
program  as  described  above.  That  is,  after  a  technician  in  the  treatment  group  finished  the  training,  researchers  conducted  a 
posttest  with  that  technician  and  the  matched  technician  in  the  control  group. 

5.6.  Social-validity  measures 

We  developed  a  14-item,  social-validity  questionnaire  to  measure  technicians’  satisfaction  with  three  major  aspects  of  the 
virtual  training  program  as  well  as  their  overall  satisfaction  with  the  program.  In  particular,  we  asked  technicians  to  provide 
ratings  about  their  experiences  with  (a)  the  web-based  technology  (i.e.,  the  process  of  using  a  virtual  meeting  platform,  the 
quality  of  the  video  and  audio  during  the  meetings  and  role-plays,  and  the  use  of  Blackboard  to  deliver  the  content  through  the 
Internet);  (b)  the  content  (i.e.,  the  breadth  of  information  covered,  the  amount  learned  from  the  modules,  the  use  of  quizzes,  and 
the  organization  of  the  modules);  (c)  their  interactions  with  the  consultant  (e.g.,  feedback  during  role-plays  and  weekly  updates; 
scheduling  flexibility):  and  (d)  their  overall  satisfaction  with  the  program  and  whether  they  would  recommend  it  to  others  who 
are  not  able  to  receive  on-site  training.  Each  item  was  scored  on  a  7-point,  Likert  scale  ranging  from  1  for  strongly  disagree  to  7  for 
strongly  agree,  with  higher  numbers  reflecting  greater  satisfaction  with  that  aspect  of  the  virtual  training  program. 

6.  Results 

6.1.  Direct-observation  measures 

We  analyzed  performance  on  the  BISPA  and  the  BISWA  assessments  using  two  scoring  methods,  one  that  calculated  the 
percentage  of  correct  trials  and  another  that  calculated  the  percentage  of  component  skills  mastered. 
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Behavioral  Implementation  Skills  Behavioral  Implementation  Skills 

for  Play  Activities  (BISPA)  for  Work  Activities  (BISWA) 


Pretest  Posttest  Pretest  Posttest 


Assessments 


Fig.  l.The  closed  circles  denote  the  average  performance  ofthe  four  technicians  randomly  assigned  to  the  (technology-enhanced)  treatment  group,  and  the 
open  circles  denote  the  average  performance  of  the  four  matched  technicians  assigned  to  the  waitlist-control  group.  The  top  panels  display  performance 
based  on  the  percentage  of  trials  implemented  correctly,  and  the  bottom  panels  display  performance  based  on  the  percentage  of  component  skills  mastered. 


6.2.  Percentage  of  trials  implemented  correctly 

With  this  method,  we  scored  the  percentage  of  trials  implemented  correctly  by  dividing  the  total  number  of  trials 
implemented  correctly  plus  the  total  number  of  errors,  including  both  errors  of  commission  (implementing  a  skill 
incorrectly:  e.g.,  delivering  reinforcement  following  an  incorrect  response)  and  errors  of  omission  (failing  to  implement  a 
skill  when  required;  e.g.,  omitting  descriptive  praise  following  a  correct  response).  This  method  of  scoring  provided  an 
overall  measure  of  how  accurately  the  technicians  carried  out  the  procedures. 

The  mean  percentage  of  trials  implemented  correctly  for  the  treatment  and  control  groups  during  pretest  and  posttest  of 
the  BISPA  are  shown  in  the  upper-left  panel  of  Fig.  1 .  Both  groups  performed  poorly  on  the  BISPA  during  the  pretest,  although 
the  treatment  group  showed  a  slightly  higher  mean  percentage  of  correct  trials  than  did  the  control  group  (Ms  =  25.8%  and 
8.3%  on  the  pretest  for  the  treatment  and  control  groups,  respectively).  However,  the  difference  between  the  means  for  the 
treatment  and  control  groups  on  the  posttest  was  notably  larger  (Ms  =  88.5%  and  1 3.3%  on  the  posttest  for  the  treatment  and 
control  groups,  respectively).  Due  to  the  observed  difference  between  the  groups  on  the  pretest,  we  analyzed  the  data  with  a 
general  linear  model  using  performance  on  the  pretest  as  a  co-variable.  The  top  panel  of  Table  3  shows  the  results  of  this 
analysis.  As  can  be  seen,  when  statistically  controlling  for  the  effects  of  the  pretest,  the  difference  between  the  treatment  and 
control  group  means  on  the  posttest  was  large  (i.e.,  partial  eta  squared  of  .765,  which  is  equivalent  to  a  Cohen’s  d  of  3.6)  and 
statistically  significant  (F=  13.0;  p  <  .05). 

Similar  results  were  obtained  for  the  percentage  of  trials  implemented  correctly  on  the  BISWA,  which  are  depicted  in  the 
upper-right  panel  of  Fig.  1.  The  treatment  group  showed  a  slightly  higher  mean  percentage  of  correct  trials  than  did  the 
control  group  on  the  BISWA  pretest  (Ms  =  50.3%  and  28.5%  for  the  treatment  and  control  groups,  respectively),  but 
the  difference  was  notably  larger  on  the  posttest  (Ms  =  97.5%  and  43.3%  for  the  treatment  and  control  groups,  respectively). 
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Table  3 

Between-groups  inferential  statistics. 


Type  III  SS 

df 

Ms 

F 

Sig. 

Partial  eta 
squared 

Noncent. 

parameter 

Observed 

power 

Mean  percentage  of  trials  implemented  correctly 

Behavioral  Implementation  of  Skills 

for  Play  Activities  (BISPA) 

Corrected  model 

n. 399.29 

3 

3799.76 

24.53 

.005 

.948 

73.59 

.990 

Intercept 

2865.87 

1 

2865.87 

18.50 

.013 

.822 

18.50 

.888 

Group 

2017.61 

1 

2017.61 

13.03 

.023 

.765 

13.03 

.769 

Pretest 

73.99 

1 

73.99 

.478 

.527 

.107 

.478 

.084 

Group  X  pretest 

24.09 

1 

24.09 

.156 

.713 

.037 

.156 

.061 

Error 

619.68 

4 

154.90 

Total 

32,725.00 

Corrected  total 

12,018.88 

Behavioral  Implementation  of  Skills 

for  Work  Activities  (BISWA) 

Corrected  model 

5957.09 

3 

1985.70 

16.73 

.010 

.926 

50.19 

.947 

Intercept 

7390.25 

1 

7390.25 

62.26 

.001 

.940 

62.26 

1.00 

Group 

1557.13 

1 

1557.13 

13.12 

.022 

.766 

13.12 

.772 

Pretest 

59.33 

1 

59.33 

.500 

.519 

.111 

.500 

.086 

Group  X  pretest 

40.60 

1 

40.60 

.342 

.590 

.079 

.342 

.074 

Error 

474.783 

4 

118.70 

Total 

46,053.00 

Corrected  total 

6431.88 

Mean  percentage  of  component  skills  mastered 

BISPA 

Corrected  model 

8318.58 

2 

4159.29 

13.10 

.010 

.840 

26.20 

.912 

Intercept 

7003.44 

1 

7003.44 

22.06 

.005 

.815 

22.06 

.959 

Group 

3854.30 

1 

3854.30 

12.14 

.018 

.708 

12.14 

.794 

Pretest 

1474.08 

1 

1474.08 

4.64 

.084 

.481 

4.64 

.415 

Group  X  pretest 

.000 

0 

.000 

.000 

Error 

1587.42 

5 

317.48 

Total 

21,156.00 

Corrected  total 

9906.00 

BISWA 

Corrected  model 

11,497.31 

3 

3832.43 

60.31 

.001 

.978 

180.93 

1.00 

Intercept 

13,472.71 

1 

13,472.71 

212.01 

.000 

.981 

212.01 

1.00 

Group 

8139.64 

1 

8139.64 

128.09 

.000 

.970 

128.09 

1.00 

Pretest 

553.40 

1 

553.40 

8.71 

.042 

.685 

8.71 

.606 

Group  X  pretest 

768.91 

1 

768.91 

12.10 

.025 

.752 

12.10 

.740 

Error 

254.19 

4 

63.55 

Total 

37,516.00 

Corrected  total 

11,751.50 

The  second  panel  of  Table  3  shows  the  results  of  statistical  analyses  for  the  percentage  of  trials  implemented  correctly  on  the 
BISWA.  When  statistically  controlling  for  the  effects  of  the  pretest,  the  difference  between  the  treatment  and  control  group 
means  on  the  posttest  was  large  (i.e.,  a  partial  eta  squared  of  .766,  which  is  equivalent  to  a  Cohen’s  d  of  3.6)  and  statistically 
significant  (F=  13.1 ;  p  <  .05). 

6.3.  Percentage  of  component  skills  mastered 

With  this  method,  we  calculated  the  percentage  of  skills  implemented  correctly  for  each  component  skill  (e.g.,  delivers 
behavior-specific  instructions)  using  the  formula  described  for  the  other  method.  Next,  we  counted  the  total  number  of 
component  skills  mastered  (i.e.,  implemented  with  90%  accuracy  or  better),  divided  that  number  by  the  total  number  of 
component  skills,  and  converted  the  resulting  proportion  to  a  percentage.  This  method  of  scoring  provided  information  on 
which  specific  skills  the  technicians  were  implementing  with  high  fidelity  and  which  ones  should  be  targeted  for  additional 
training. 

The  bottom-left  panel  of  Fig.  1  shows  the  mean  percentage  of  component  skills  mastered  for  the  BISPA  across  the 
treatment  and  control  groups.  Both  groups  displayed  mastery  performance  on  a  small  percentage  of  the  skills  (Ms  =  8.3% 
and  0%  on  the  pretest  for  the  treatment  and  control  groups,  respectively).  By  contrast,  the  treatment  group  had 
mastered  the  majority  of  the  component  skills  by  the  posttest,  whereas  the  control  group  showed  little  Improvement 
(Ms  =  66.8%  and  8.3%  on  the  posttest  for  the  treatment  and  control  groups,  respectively).  The  third  panel  of  Table  3 
shows  the  results  of  the  statistical  analyses  based  on  the  mean  percentage  of  component  skills  mastered.  When 
statistically  controlling  for  the  effects  of  the  pretest,  the  difference  between  the  treatment  and  control  group  means  on 
the  posttest  was  large  (i.e.,  partial  eta  squared  of  .708,  which  is  equivalent  to  a  Cohen’s  d  of  3.1)  and  statistically 
significant  (f=12.1;  p  <  .05). 
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The  bottom-right  panel  of  Fig.  1  depicts  the  results  for  the  mean  percentage  of  component  skills  mastered  on  the  BISWA. 
Both  groups  displayed  mastery  performance  on  a  small  percentage  of  the  component  skills  (Ms  =  25.0%  and  6.8%  on 
the  pretest  for  the  treatment  and  control  groups,  respectively).  By  contrast,  the  treatment  group  had  mastered  almost  all  of 
the  component  skills  by  the  posttest,  whereas  the  control  showed  only  slight  improvement  (Ms  =  93.3%  and  22.5%  on  the 
posttest  for  the  treatment  and  control  groups,  respectively).  The  bottom  panel  of  Table  3  shows  the  results  of  the  statistical 
analyses  conducted  on  the  mean  percentage  of  component  skills  mastered.  When  statistically  controlling  for  the  effects  of 
the  pretest,  the  difference  between  the  treatment  and  control  group  means  on  the  posttest  was  large  (i.e.,  partial  eta  squared 
of  .97,  which  is  equivalent  to  a  Cohen’s  d  of  11. 4)  and  statistically  significant  (f=  128.1  :p  <  .001).  It  is  also  noteworthy  that  on 
this  measure  the  effects  of  the  pretest  were  statistically  significant  (f  =  8.7;  p  <  .05),  as  was  the  interaction  between  pretest 
performance  and  group  membership  (f  =  12.1 ;  p  <  .05).  These  results  suggest  that  technicians  who  scored  relatively  higher 
on  the  pretest  also  tended  to  score  relatively  higher  on  the  posttest.  Finally,  the  significant  interaction  between  the  pretest 
and  group  membership  suggests  that  the  technicians  in  the  treatment  group  who  scored  relatively  higher  on  pretest  tended 
to  show  a  better  response  to  the  treatment. 

7.  Social-validity  measures 

The  mean  satisfaction  rating  with  the  web-based  technology  was  5.4  (range,  4.5-6),  with  the  least  satisfaction  expressed 
for  the  use  of  Blackboard  as  the  medium  for  accessing  the  e-learning  modules  and  the  most  satisfaction  expressed  for  the 
audio  and  visual  quality  during  role-play  sessions.  The  mean  satisfaction  rating  with  the  content  of  the  virtual  training 
program  was  6.8  (range,  6.8-7),  with  high  satisfaction  expressed  for  all  aspects  of  the  training  content,  including  the  type  of 
information  covered,  the  role-play  component,  the  quality  and  organization  of  the  e-learning  modules,  and  the  amount 
learned  during  training.  The  mean  satisfaction  rating  for  the  interactions  with  the  researcher  (behavioral  consultant)  was  6.9 
(range,  6.8-7),  with  high  satisfaction  expressed  for  all  aspects  of  the  interactions,  including  the  consultant’s  ability  to  answer 
questions,  the  option  of  completing  the  training  at  the  technician’s  pace,  and  the  flexibility  offered  in  scheduling  meetings 
and  role  plays.  Finally,  the  mean  rating  for  overall  satisfaction  with  the  training  was  7.0,  and  all  of  the  technicians  indicated 
that  they  would  recommend  the  training  to  others. 

8.  Discussion 

The  first  eight  participants  enrolled  in  an  ongoing  RCT  to  test  the  effects  of  a  virtual  training  program  for  training  ABA 
technicians  were  randomly  assigned  to  either  the  treatment  or  the  control  group.  Technicians  in  both  groups  performed 
relatively  poorly  during  the  pretest  with  the  direct-observation  measures  (the  BISPA  and  BISWA),  both  when  we  analyzed 
the  percentage  of  trials  implemented  correctly  and  the  percentage  of  component  skills  mastered.  Following  exposure  to  the 
virtual  training  program,  technicians  in  the  treatment  group  showed  marked  and  statistically  significant  improvement  on 
these  direct-observation  measures  during  the  posttest  relative  to  the  control  technicians  who  did  not  receive  the  training.  In 
addition,  social-validity  ratings  were  uniformly  high  for  the  content  of  the  training  and  the  interactions  with  the  behavioral 
consultant  and  were  moderately  high  for  the  web-based  technology  (e.g.,  GoToMeeting).  Finally,  all  technicians  who 
received  the  training  rated  their  satisfaction  with  the  programs  at  the  top  of  the  Likert  scale  (i.e.,  all  rated  it  as  a  7  on  a  7-point 
scale),  and  all  technicians  indicated  that  they  would  recommend  the  training  to  others. 

Several  studies  have  evaluated  similar  virtual  programs  for  training  parents  or  technicians  to  implement  behavioral 
interventions  for  children  with  ASD  (Buzhardt&FIeitzman-Powell,  2005;  Flamad  et  al.,  2010;  Jang  et  al.,  2012;  Vismara  et  al., 
2009).  The  current  results  replicate  and  extend  the  findings  of  those  prior  studies  in  several  ways.  First,  the  prior  studies  and 
ours,  taken  together,  show  that  virtual  technologies  can  be  used  to  provide  effective,  convenient,  accessible,  and  socially- 
valid  training  in  EIBI  or  other  early  intervention  methods.  In  the  current  study,  the  technicians  in  the  treatment  group 
completed  the  e-learning  modules  on  their  own,  at  their  own  pace,  and  in  their  own  home  or  community  environment.  This 
component  of  the  training  required  little  time  or  effort  on  the  part  of  the  researchers  (after  the  development  of  the  e-learning 
modules  was  completed).  Obviously,  the  role-play  sessions  required  active  participation  from  a  researcher,  but  overall,  the 
virtual  training  program  was  reasonably  time  efficient.  In  addition,  neither  the  technicians  nor  the  researchers  had  to  travel 
in  order  to  implement  the  virtual  training  program,  thus  saving  both  time  and  money.  Therefore,  the  current  virtual  training 
program  could  potentially  be  used  to  train  ABA  technicians  anywhere  there  is  sufficient  broadband  Internet  access. 

Second,  only  one  prior  study  has  evaluated  a  virtual  training  program  using  an  RCT  Qang  si-.  2012),  and  that 
investigation  targeted  knowledge  of  behavioral  procedures  as  measured  by  a  multiple-choice  quiz.  Technicians  in  the 
current  study  completed  a  40-h  virtual  training  program  designed  to  teach  knowledge  about  ABA  principles  and  procedures 
as  well  as  train  the  technicians  to  implement  a  variety  of  behavioral  protocols  with  high  procedural  integrity.  The  technicians 
in  the  treatment  group  were  required  to  obtain  a  score  of  80%  correct  or  better  on  multiple-choice  quizzes  administered  at 
the  completion  of  each  e-learning  module,  and  they  received  scripted  role-play  practice  with  feedback  in  implementing 
behavior-reduction  and  skill-acquisition  procedures  implemented  in  both  discrete-trial  and  play-based  formats.  Thus,  the 
current  virtual  training  program  was  considerably  more  comprehensive  than  most,  if  not  all,  of  the  virtual  training  programs 
evaluated  in  prior  studies.  In  fact,  after  completion  of  the  training,  all  four  technicians  met  the  qualifications  to  work  as  a 
behavioral  tutor  under  the  rules  of  the  Tricare  Autism  Demonstration  Project  (TriCare  Operations  Manual  -  6010.56-M, 
Chapter  18,  Section  8,  C-101,June  25,  2013). 
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Another  unique  and  potentially  interesting  aspect  of  the  current  study  was  that  behavioral  skills  training  (BST) 
procedures  used  during  the  role-play  sessions  were  completed  without  any  on-site  support  from  a  trained  professional  or 
technician.  BST  is  a  widely  used  method  for  training  individuals  to  implement  ABA  procedures  that  has  strong  empirical 
support  (see  Thompson,  Martin,  Arnal,  Fazzio,  &  Yu,  2009  for  a  review).  The  four  primary  components  of  BST  are  instruction, 
modeling,  rehearsal,  and  feedback.  These  training  procedures  have  been  almost  always  Implemented  with  the  trainer  and 
trainee  at  the  same  physical  location. 

Wacker  and  colleagues  have  pioneered  the  use  of  similar  procedures  to  train  and  guide  professionals  to  conduct 
controlled  functional  analyses  at  remote  sites  using  fiber-optic  telecommunications  system  (Barreto,  Wacker,  Harding,  Lee, 
&  Berg,  2006)  and  VPNs  (Wacker  et  al.,  2013a,  2013b).  However,  their  procedures  have  included  trained,  on-site  assistants 
that  provided  support  to  the  parents  who  implemented  the  functional  analyses  or  functional  communication  training  in  a 
hospital  or  clinic  located  near  the  family.  In  addition,  procedural  integrity  data  were  not  reported;  as  a  result,  the  extent  to 
which  the  parents  implemented  the  procedures  with  fidelity  remains  uncertain.  By  contrast,  in  the  current  study,  all 
components  of  BST  were  conducted  remotely  using  a  VPN,  and  at  posttesting,  the  technicians  in  the  treatment  group 
generally  Implemented  the  procedures  accurately  (i.e.,  97.5%  of  trials  implemented  correctly  on  the  BISWA,  and  88.5%  of 
trials  Implemented  correctly  on  the  BISPA).  Similarly,  the  confederates  also  performed  their  activities  with  high  procedural 
integrity,  probably  because  the  behavioral  consultant  verbally  guided  the  confederates  as  they  played  the  role  of  a  child  with 
an  ASD  (I.e.,  94.4%  of  trials  were  correctly  programmed  for  the  BISWA,  and  99.8%  of  trials  were  correctly  programmed  for  the 
BISPA). 

These  findings  should  be  regarded  as  preliminary  and  interpreted  with  appropriate  caution,  given  the  limitations  of  the 
investigation.  We  reported  on  only  the  first  eight  technicians  enrolled,  and  only  four  of  these  technicians  received  the 
treatment.  A  related  limitation  is  that  the  treatment  technicians,  as  a  group,  scored  higher  on  the  pretest  measures  than  did 
the  control  technicians,  which  can  occur  in  RCTs  when  the  sample  sizes  are  small.  We  controlled  for  this  difference 
statistically  In  our  analyses,  but  on  one  of  the  measures  we  observed  an  interaction  between  the  pretest  and  group 
membership,  suggesting  that  individuals  with  higher  pretest  scores  tended  to  respond  better  to  the  intervention.  Thus,  it  is 
possible  that  the  effect  size  would  have  been  somewhat  smaller  for  this  dependent  measure  if  the  groups  had  been  better 
equalized  at  pretest.  As  we  continue  to  enroll  technicians,  we  will  use  the  process  of  minimization  to  produce  more  equal 
groups  going  forward.  That  Is,  technicians  will  be  assigned  one  at  a  time  with  an  80%  chance  of  being  assigned  to  the  group 
that  would  minimize  the  Kullback-Lelbler  divergence  index  using  the  MinimRan  virtual  program  with  the  pretest  scores  on 
the  BISWA  and  BISPA  as  the  co-varlables  (Xiao,  Huang,  Yank,  &  Jun  Ma,  2013).  This  procedure  should  produce  equivalent 
means  for  the  pretest  measures  In  the  final  report  of  this  ongoing  investigation. 
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